Time delay and integrate (TDI) cameras are operated in sensors where low-light levels and/or high-speed image motion preclude achieving adequate resolution with a snapshot-mode camera. See, e.g.: Farrier, M. and Dyck, R., “A Large Area TDI Image Sensor for Low Light Level Imaging,” IEEE Trans. Electron. Dev., Vol. ED-27, No. 8 (1980); and Wong, H.-S., Yao, Y. L., Schlig, E. S., “TDI Charge-coupled devices: Designs and applications,” IBM J. Res. Develop, Vol. 36, No. 1 (1992).
Reconstruction of a high-resolution image from a sequence of lower resolution images is a way to increase the effective spatial resolution of a camera capturing conventional movie images. See, e.g.: Huang, T. S. and Tsai, R. Y., “Multi-frame image restoration and registration,” Advances in Computer and Image Processing, Vol. 1, (1984); and Borman, S. and Stevenson, R. L., “Super-resolution from image sequences—A review,” Proc. 1998 Midwest Symp. Circuits and Systems, pp. 374-378 (1999).
The simplest reconstruction schemes use a simple interlace of the data to form the high-resolution image. See, e.g.: Grycewicz, T. J., Cota, S. A., Lomheim, T. S., and Kalman, L. S., “Focal plane resolution and overlapped array TDI imaging,” Proc SPIE 708703 (2008); Watson, E. A., Muse, R. A., and Blommel, F. P., “Aliasing and blurring in microscanned imagery,” Proc. SPIE 1689 (1992); and Gilette, J. C., Stadtmiller, T. M., and Hardie, R. C., “Aliasing reduction in staring infrared imagers using subpixel techniques,” Optical Engineering, Vol. 34, No. 11 (1995). More robust schemes use a shift-and-add technique. See, e.g.: Farsiu, S., Robinson, D., Elad, M., and Milanfar, P., “Robust Shift and Add Approach to Super-Resolution,” Proc. SPIE 5203, pp. 121-130 (2003). The most accurate algorithms use iterative algorithms such as those based on gradient descent or projection onto convex sets (POCS) to optimize the reconstruction for each image set. See, e.g.: Kim, S. P., Bose, N. K., and Valenzuela, H. M., “Recursive Reconstruction of High Resolution Image From Noisy Undersampled Multiframes,” IEEE Trans Accoustics, Speech, and Signal Processing 38(6), 1990; Tom, B. C., Katsaggelos, A. K., “Reconstruction of a high resolution image by simultaneous registration, restoration, and interpolation of low-resolution images,” Image Processing, 1995, Proceedings, International Conference on (1995); Schultz, R. R. and Stevenson, R. L., “Extraction of High-Resolution Frames from Video Sequences,” IEEE Trans. Signal Processing 5(6), 1996; and Matson, C. L. and Tyler, D. W., “Primary and secondary super-resolution by data inversion,” Optics Express, Vol. 13, No. 2 (2006). Real-time applications have generally used the interlace reconstructors. See, e.g.: Alam, M. S., Bognar, J. G., Hardie, R. C., and Yasuda, B. J., “Infrared Image Registration and High-Resolution Reconstruction Using Multiple Translationally Shifted Aliased Video Frames,” IEEE Trans. Instrumentation and Measurement, Vol. 49, No. 5 (2000).
NASA's Drizzle algorithm applies super-resolution techniques to reconstruct images taken with the wide-field cameras on the Hubble Space Telescope. See, e.g., Fruchter, S. A. and Hook, R. N., “Drizzle: A Method for the Linear Reconstruction of Undersampled Images,” PASP 114:144-152 (2002). See also: U.S. Pat. No. 5,341,174 to Xue et al., entitled “Motion Compensated Resolution Conversion System”; U.S. Pat. No. 5,696,848 to Patti et al., entitled “System for Creating a High Resolution Image from a Sequence of Lower Resolution Motion Images”; U.S. Pat. No. 5,920,657 to Bender et al., entitled “Method of Creating a High Resolution Still Image Using a Plurality of Images and Apparatus for Practice of the Method”; U.S. Pat. No. 6,023,535 to Aoki, entitled “Methods and Systems for Reproducing a High-Resolution Image from Sample Data”; U.S. Pat. No. 6,208,765 B1 to Bergen, entitled “Method and Apparatus for Improving Image Resolution”; U.S. Pat. No. 6,535,650 B1 to Poulo et al., entitled “Creating High Resolution Images”; U.S. Pat. No. 7,085,323 B2 to Hong, entitled “Enhanced Resolution Video Construction Method and Apparatus”; and U.S. Pat. No. 7,352,919 B2 to Zhou et al., entitled “Method and System of Generating a High-Resolution Image from a Set of Low-Resolution Images”.
Super-resolution reconstruction has been used with line scan and TDI imagers where the focal plane consists of two imaging arrays with a sub-pixel offset between the pixel locations in one array and the locations in the other, as shown in FIG. 1. See, U.S. Pat. No. 7,227,984 B2 to Cavan, entitled “Method and Apparatus for Identifying the Defects in a Substrate Surface by using Dithering to Reconstruct Under-Sampled Images” and Grycewicz et al.
The overlapped array scheme has been implemented for the 2.5 m GSD “supermode” on the ESA SPOT-5 imaging satellite. See, Jacobsen, K., “High-Resolution Imaging Satellite Systems,” 3D-Remote Sensing Workshop, Porto (2005), accessed at http://www.ipi.uni-hannover.de/uploads/tx_tkpublikationen/HRIjac.pdf and Poon, J., Smith, L., and Fraser, C., Orthoimage Resolution and Quality Standards, Project Number 2.3 Final Report, CRC for Spatial Information, University of Melbourne (2006).
As discussed in Grycewicz et al., overlapped (i.e., staggered) time delay and integrate (TDI) scanning arrays with interlaced columns can provide up to twice the effective resolution of conventional TDI focal plane arrays with the same pixel size when operated under nominal conditions. However, especially when the overlapped TDI arrays are physically separated on the camera focal plane, image drift can destroy the alignment that allows for super-resolution reconstruction of the overlapped images. Even small amounts of uncompensated jitter or image drift have been shown to completely destroy the resolution improvement gained from interlacing offset array data. See, e.g.: Grycewicz et al.; Hochman, G., Yitzhaky, Y. Kopeika, N. S., Lauber, Y., Citroen, M., and Stern, A., “Restoration of Images Captured by a Staggered Time Delay and Integration Camera in the Presence of Mechanical Vibrations,” Applied Optics, Vol. 43, No. 22, pp. 4345-4354 (2004); and Haik, O. and Yitzhaky, Y., “Superesolution reconstruction of a video captured by a translational vibrated staggered TDI camera,” Proc. SPIE 5558, pp. 815-826 (2004). See also, Young, S. S. and Driggers, R. G., “Superresolution image reconstruction from a sequence of aliased imagery,” Applied Optics, Vol. 45, No. 21, pp. 5073-5085 (2006).
It would be useful to be able to decrease the susceptibility of overlapped TDI arrays (or other optical detectors arranged with overlapping fields of regard) to loss of high-resolution performance in the presence of image drift. It would be useful to be able to improve the tolerance of overlapped-array imaging technologies to scan rate errors. It would be useful to be able to minimize (or decrease) the conditions under which image drift results in severe degradation of the resolution gain potentially realized from the use of staggered arrays. It would be useful to be able to utilize multiple imaging arrays while assuring sufficiently good pixel coverage for super-resolution image reconstruction. It would be useful to be able to utilize three or more overlapped imaging arrays while assuring good performance with moderate image drift. It would be useful to be able to optimize (or enhance) the resolution of images formed from staggered array outputs and, in particular, from three or more staggered array outputs.